Skip to content

bench(index): refresh streaming memory study on current dedup - #215

Open
adarshh347 wants to merge 2 commits into
StarTrail-org:mainfrom
adarshh347:codex/index-build-memory-study-rebase
Open

adarshh347 wants to merge 2 commits into
StarTrail-org:mainfrom
adarshh347:codex/index-build-memory-study-rebase

Conversation

@adarshh347

Copy link
Copy Markdown
Contributor

Related to #179. This is an experimental benchmark and study only; it does not change production behavior, CLI defaults, dependencies, or lockfiles.

The study was originally based on b4fe763. Current upstream includes #203, which replaced the old fixed-decimal dedup key with range-derived identity and rejects ranges that do not fit in int64. The rebase updates the prototype to use an on-disk SQLite composite key, preserves first-wins ordering, and checks the same range limit. Tests cover the former key-collision pairs and the current range-rejection behavior.

Validation:

  • The expected baseline allocation regression fails against the current production builder at the full (384, 16) vector matrix allocation.
  • Focused and neighboring index/serving suites: 48 passed, 0 skipped.
  • Four synthetic benchmark configurations completed with three fresh-process builder pairs each; raw reports are included.
  • All five metadata arrays, dtypes, and row order matched the production merge in all configurations.
  • git diff --check passed.
  • Ruff could not be run locally: it is not installed, and DNS prevented fetching it. Full browser/ML tests were not run.

At 65,536 × 256 generated rows, median peak RSS changed from 272.73 to 144.52 MiB for IVFFlat and from 264.75 to 76.23 MiB for IVFPQ. These short shared-machine synthetic measurements do not establish production-scale behavior or semantic-retrieval quality. The final IVFFlat index still occupies memory; Linux and representative-data evaluation remain unvalidated.

Add an isolated streaming prototype, real-module regression tests, and reviewed synthetic benchmark reports. Preserve production defaults and legacy dedup semantics; document ANN sampling and production-design limitations.

Proposal: StarTrail-org#179
@vercel

vercel Bot commented Nov 18, 2026

Copy link
Copy Markdown

@adarshh347 is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant